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Researchers have become concerned with the need 
make research relevant to the interests and concerns of 
practitioners^ and to facilitate the application of new knowledge 
about teaching and learning to the ongoing teaching-learning process 
of the classroom. This paper presents examples of the ways in which 
research findings from two ETEP (Effective Teacher Education Program) 
studies and two data collection procedures taken from these studies 
may be applied to teaching and to the training of teachers. The two 
ETEP studies concern the effects of teacher use of probing and 
redirection and teacher use of higher cognitive questions on student 
achievement and attitudes. The four approaches to application of the 
research are: (1) application through development of teacher training 
programs and materials; (2) application through use of new knowledge 
by teachers to modify how they structure the teaching/learning 
situation and how they interact with students; (3) application 
through use of data collection procedures as instructional vehicles 
and as self-development tools; and (4) application through teachers 
serving as researchers who validate and evaluate findings as they are 
applied in the classroom, A consideration of a new approach to 
research and development in teaching^ which will increase the 
likelihood that research on teaching will be applied in the classroom 
in order to improve educational opportunities^ is presented, (JMF) 
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From its inception nine years ago, the Far West Laboratory for 
Educational Research and Development has been engaged in research and 
development activities related to teaching. This work began with heavy 
emphasis upon application of research to teaching through the develop- 
ment of teacher training products. More recently, our work has emphasized 
basic research seeking answers to the question, "What is effective teach- 
ing in specific instructional situations for what types of learners?" 

we have become more involved with conducting rather than applying 
research, we also have become concerned with the need to make the research 
relevant to the interests and concerns of practitioners^ i.e., teachers 
and teacher train :rs, and to facilitate the application of new knowledge 
about teaching and learmng to the ongoing teaching-learning process of 
the classroom. 

The purpose of this paper is to explore several ways in which 
research can be applied to teaching. For purposes of illustration, 

* Work reported herein was supported by the National Institute of 
Education, Department of Health, Education, and Welfare, Contract 
No. NE'C-00-3-0103. The opinions expressed in this publication do 
not necessarily reflect the position or policy of the National 
Institute of Education and no official endorsement by the Institute 
should be inferred. 
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research findings from three studies conducted by the Effective Teacher 
Education Program (ETEP)* v/ill be presented. 



Application of Research to Teaching: What Do We Mean? 

The research process may be categorized into five major steps: 

1. Establishing the questions to be answered/hypotheses to h2 
tested. 

2. Preparing a research design that attends to these qiestions 
including Identifying the variables of interest; identifying/ 
developing procedures for studying (measuring) these variables; 
and determining what, if any, treatment conditicis are to be 
imposed. 

3. Conducting the study. More specifically, this involves 
setting up the conditions necessary to collect the desired 
information (data) and collecting the information itself. 

4. Analyzing the data so as to obtain answers to the questions/ 
hypotheses on which the research is focused. 

5. Reporting and interpreting the results. 

Given the effects of each of these steps upon the educational know- 
ledge base, it can be argued that all five steps are relevant to teaching. 
We agree: teachers and teacher trainers should be involved in the entire 
process. However, in terms of direct applicability to the ongoing 
teaching process, the two aspects of research that are most important 
are the findings that result from the research (the new knowledge that 
1s gained) and the methods and procedures used to acquire the new infor- 
mati bout teaching and learning. 



* A research program carried out at the Far West Laboratory under the 
auspices of NIE Contract No. NE-C-3-0i08. The research studies include: 
Gall, et al., Th e Effects of Teacher Use of Questioning Techniques On 
Stude nt A chiev^?inent and At titudes; Snow, et al>. Extended Analysis of 
JviO Ex perinont s on Teac hing; Wiiriams, M ath Tutoring Study: Teachers 
as Tutors ; Paraprofessionals as Tutors ; Cross-age Tutoring ^ 
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It is not necessary here to elaborate the need to oply new knowledge 
to teaching. This topic has long been a matter of concern to the field of 
education. 

On the other hand, application of the research methods and procedures 
ws'-* to acquire inform:.tion about teaching and learning is a relatively 

•^e most part, practitioners have not capitalized 
ting that goes into developing these pro- 
■ection are seldom applied to the 
ess frequently utilized directly in 

the teach! ./learn;. 

A more definitive ourpu.if ot thi<; paper, therefore, is to present 
examples of: 

- the ways in which research findings froin two ETEP studies may be 
applied to teaching and teacher training; and 

- the ways in which two data collection procedures taken from these 
studies may be applied to teaching and to the training of teachers. 

The findings and procedures to be discussed are exemplary only. They 
in no way exhaust the array of new knowledge resulting from the research 
and/or the complete set of data collection procedures used in the various 
ETEP studies. 

In addition to discussing the findings and data collection procedures 
from the ETEP research on teaching, the paper concludes by considering 
a new approach to research and development in teaching— an approach that 
we propose will increase the likelihood that research on teaching will 
be applied in the classroom in order to improve educational opportunities- 

*>flicroteaching and Flanders' Interaction Analysis systems are two examples 
when this has been done. 
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Four Approaches to Application of Research 

The application of research to teaching may iplished in a 

variety of ways. Four strategies thr t appear to be rkable are discussed 
below. 

^ • Application through development of teacher training programs 
and materials. Training programs and materi \ls have been the primary 
approach to applying research to teaching for the past ten yeeirs. A 
common theme for educational reform in the late ^60's and early 1970' s 
was the need for products that would promote the use of new knowledge 
and skills by teachers. Inasmuch as the majority of the individuals in 
education today are familiar with one or more of these products, this 
paper will not elaborate on the characteristics of this application 
strategy. 

2- Application through use of new knowledge by teachers to modify 
how they structure th e teaching/learning situation and how they interact 
with students. The ultimate goal of any educational research is appli- 
cation of new knowledge in the ongoing learning experiences of each 
student. With or without special training, a teacher applies knowledge 
about teaching and learning whenever (s)he makes an instructional decision. 
The challenge, therefore, is to make research findings available to 
teachers in a form and via a process that relates directly to classroom 
dcuision-making. This paper will give special consideration to this 
strategy. 

3. Application through use of data collection procedures as 
in structional vehicles and as self-development tools . In order to gain 
control of the ccntent and, at times, the process of instruction, 
reijearchers frequently develop curriculum units, tests and/or other 
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techniques for measuring student performance, systems for observing and 
recording teacher-student interactions, etc. A teacher can utilize such 
materials to improve instruction within the classroom. 

For example, a fifth grade teacher might incorporate into a regular 
mathematics program those curriculum materials that were originally 
developed for a research study of effective mathematics teaching at that 
grade level. (S)he also might adopt for classroom use a diagnostic 
instrument used to measure student reading comprehension in a research 
study of the teaching of reading. 

This paper will present three examples of research procedures that 
may be applied in the classroom. 

A teacher also can apply data collection procedures as self-analytic 
tools. The interaction analysis system .and microteaching are two examples 
of data collection procedures that have later been used by teachers as a 
means for analyzing and improving their teaching. Two others will b 
discussed in this paper. 

4. Application through teachers serving as researchers who vali- 
date (and evaluate) findings as they are applied in the classroom. 
Replication of a particular research study is the generally accepted 
procedure for establishing that a given set of findings is valid and 
stable. Another approach that may be less rigorous, but that nonetheless 
is empirically sound, would be to have teachers apply and test thG bind- 
ings in their instructional programs. Immediately following each application, 
the teacher would report the outcomes from both a teacher and student 
perspective. Compilation of such reports would provide valuable evidence 
of the accuracy and worth of the research findings. It also would siiggest 
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new areas of inquiry that should be pursued by the researchers. This 
paper also will provide an example of this approach to application of 
research in teaching. 



Jhft EffectivQ Teacher Education Program 

The Effective Teacher Education Progran (ETEP) is a program of 
research and development in teaching that, to date, has had three major 
purposes 

- To develop teacher training materials that incorporate the micro- 
teaching approach to training and that develop teachers' use of 
specified teaching skills (the Minicourses) ; 

- To study the effects upon student outcomes of specific teaching 
skills (for the most ;art these have been skills that were con- 
tained in one or more of the Minicourses); 

- To test various approaches to the study of teaching. 

The current programmatic effort is devoted entirely to the latter 
two objectives. Some two and a half :'ears of research are now completed 
and in the final stages of report preparation. 

The research topics selected for study by ETEP were a direct out- 
growth of the competency-baied teacher education movement. Teaching 
skills considered important by theorists, researcfv "s, and trainers 
(e.g., Dodl, 1972; Turner, et al,, 1973) were included in the Minicourses. 
These skills, in turn, became the focus of the ETEP research. The three 
aspects of teaching that have been investigated include: 

- Use of questioning skills . Two questioning studies were conducted. 
The purpose of Study I was to study the effects on student achieve- 
ment and attitudes of teacher use of probing and redirection during 
a discussion. The purpose of Study 11 was to investigate the effects 
of teacher use during discussions of differing proportions of higher 
cognitive questions (25:^, 50^., 75%) in relation to fact-recall ques- 
tions on student achievement and attitude. The teaching skills of 
interes. were taken from Minicourse T, Effective Questioning - 
Elementary Level , and Minicourse 9, Higher Cognitive Questioning . 
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" of mathematics tutoring ski 1 U . Three studies were conducted 
each employing a ditferent'individual in the tutori^lrofe (regular 
classroom teacher, paraprofessional . junior high school student). 

Sn^nnrr^^J^-^^^^^^" " ^^^^^ "i 1 1 be presented here. The 

purpose of this study was to answer questions about the effects of 

conce';t"'arirrela'Ji'to°" T'''.''' "'^^^^"'^tics achievement! se?f- 
concept as it relates to mathematics, attitude toward mathematics 

'tuloHn'o'sk 1 fJo Si TT'l " ''''''' ^° n,athema??cs ' ? e • 
tutor ng ski Is to be studied were taken from Minicourse 5, 
Individualizi ng Instruction in Mathematics . 

~ of an independent l earning system . The purpose of th^s studv w;,c 
to investigate the effect upon students of an instructional sJstem 
Training in the implementation and use of the Sendent ea^ninS' 
iJIt "ed'lte' ui^r' Vd^rriV''-. grganizinT?nd"epen'denrLe;gn in,. 

In addition to studying the effects on students of teacher use of 
various skills, the ETEP research has explored four approaches to the 
study of teaching. The approaches fall along a continuum of research 
techniques that range from a tightly controlled laboratory experiment to 
work in the regular classroom where instructional and other variables 
are allowed to vary naturally (see Figure 1). 



FIGURE 1 

Continuum of Approaches to the Study of Teaching 

' • I Classroom 



1 — Semi -programmed approach 

2- -Experimental Teaching Unit 

^■^r-3'n Ittr^T' ^^'t'^L^^t'''-^'': ^^'^-^^ ^"^ specified instructional setting 
4-Train teacher; establish instructional system; study overall effect. 
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The semi -programmed approach was used in che questioning studies. 
In this approach, the teacher is provided a set of curricular materials 
to use with the students. The sequence in which the materials are to be 
used is specified and the teacher is directed to conduct discussion and/or 
some other activity as part of each day's lesson. When a discussion is 
specified, the teacher is provided a script which tells her/him the 
questions to ask and in what sequence. The script is only "semi" programmed 
because some teacher behaviors during a discussion are contingent 'jpon 
student responses. For example, in the questioning studies, the script 
could dictate the question to be asked and approximately how many times 
probing was to occur but it could not prescribe which student responses 
would be probed. 

The Experimental Teaching Unit (ETU) approach was incorporated as 
a sub-study in the mathematics tutoring study with teachers as tutors. 
This unit consisted of a statement of teaching and learning objectives, 
curricular materials for students, and criterion-referenced pre and post 
tests. Teachers were allowed to organize and teach the unit as they 
wished. This approach, therefore, controls the content but not the 
process of instruction. 

The third approach, training teachers to use a particular set of 
skills, then studying the effects in a given content area and/or instruc- 
tional setting, is a familiar form of research on teaching. It has been 
used in a large number of the existing studies of teachers. It was 
employed in the ETEP math tutoring studies. 

The fourth approach parallels the research procedures applied to 
ma. previous studies of innovative educational programs. Teachers 
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are trained in a new educational pngram; the program is put into opera- 
tlor?; the effects on students are studied. As noted earlier, this 
approach was used in the ETEP independent learning study. 

The discussion which follows is concerned with reporting and apply- 
ing the findings of the ETEP Questioning and Teacher as Math Tutor 
Studies; and with considering the usefulness and applicability of the 
semi -programmed and ETU approaches to research on teaching. The Indepen- 
dent Learning Study will not be discussed inasmuch as data analyses are 
not yet completed. 



The Questioning Studies 

Two studies of teacher use of questioning skills have been conducted 
as part of the ETEP effort. They are Stu«.ly I: The Effects of Teacher 
Use of Probing and Redirection on Student Achievement and Attitudes and 
Study II: The Effects of Teather Use of Higher Cognitive Questions on 
Student Achievement and Attitudes .* Both studies were directed by Dr. 
Meredith D. Gall, who has recently joined the School of Education faculty 
at the University of Oregon. 

Both Study I and Study II were designed to reduce imd/or eliminate 
several previously identified research problems. These included the need 
for random assignment of students to treatment, matching of student 

* For complete report of the studies see: Gall, et al.. The Effect *; 
tlJt^^J^^^ °^ Qu estioning Techniques on Student Achievement and 
Attitudes. San Francisco: Far West Laboratory for Educational 
Research and Development, 1975. 
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outcone masures with tht Intended Instructional purpose(s) of the trait- 
■sent conditions, controlling for opportunity to learn, control and 
■onltorlng of treatnent conditions to be sure that/ were Maintained, and 
Investigation of the critical assumptions underlying the statistical 
procedures used <n analyses of data. 

Study I was designed to determine what student learning outeeaM mr% 
affected by presence or absence cf probing and redirection f«i dlsoistlMt 
and presence or absence of discussions thenselves. It alto Inrntlfattd 
the relative effect on student learning of teachers' questleni dtllverttf 
In discussions coMpared with the sane questions presented and an s w a rni 
In irnten foroat. 

While the outcoaws of Study I are of Interest, we have elected te 
devote this presentation to a discussion of Study II. Complete InfiOfVitiM 
regarding Study I is available In the research report. 

Study n. The Effects of Teacher iise of Kloher Cognitive Questions , 
on Student Achievement Attitudes . Included four treatment conditions- 
three discussion trealanents and an art activity treatment. All treitftiiil 
were administered to students randomly formed Into treatment groups fjreai 
sixth grade classrooms In the cooperating school district. All groups 
used a specially prepared curriculum on ecology. The trtataents were 
a^lnlstered by specially trained teachers, not the students* regular 
teachers. Uct^ teacher taught four different treatment groups each 
using the specially prepared currlcular materials and the seml-progrwnd 
discussion script to guide the presentation of the treatments. Table 1 
outlines these treatments and Indicates the percent of higher cogiiltlve 
qoestlons and the other teaching skills used In the discussions base^ on 
the sem1-progra«nd scripts. The treatments covered ten lessons In ecoltv. 

12 
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TABLE 1 




oTUuY 1 - 1 rea imenis 


Treatment 


Lon tent 


25% Higher Cognitive 
Questions 


Curriculum materials followed by discussion 
including 8F + 4MF + 4HCQ + probing and r'2- 
direction . 


50% Higher Cognitive 


Curriculum materials followed by discussion 
inrlijflinn dF + /IMF + ftHCO + Drobino'aTid 
redirection. 


75% Higher Cognitive 
Questions 


Curriculum materials followed by discussion 
including 4MF + 12HCQ + probing and 
redirection. 


. Art Activity 


Ecology - related art activity 



F = Fact question 
MF = Multi-fact question 
HCQ = Higher cognitive question 



Fidelity of treatment was investigated along two dimensions. First, 
a check was made on how closely the teachers adhered to the semi -programmed 
scripts. Second, the time required to conduct the lessons in each treat- 
ment was checked. Both these factors were checked by audiotaping three 
of the discussions conducted by each teacher on each of two days (Lesson 2 
and Lesson 7). This sample of six discussions per teacher was rated to 
establish treatment fidelity. . .. 

According to the ratings of the audiotapes, the teachers appear to 
have followed the question format. In only six instances out of the 144 
lessons rated for Studies I and II combined did teachers change the order 
of the questions. Other deviations either did not occur or occurred very 
Infrequently. j[3 
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The treatments were similar in length. The average length per treat- 
ment was 16.24 minutes for the 25% HCQ treatment plus 12 minutes of filler 
activity; 21.87 minutes for the 50% HCQ treatment plus 6 minutes of filler; 
and 27.65 minutes of discussion for the 75% HCQ treatment. The art activity 
treatment stopped after 29 minutes each day. 

The effects of the Study II treatments were investigated through a 
"Lcrttff ;iquare" design. The term "Latin square" comes from an ancient 
puzzle that deals with the number of different ways Latin letters can be 
presented in a square (matrix) such that each letter appears once, and 
only once, in each column and each row. Thus, in Study II a Latin square 
assignment as presented in Table 2 was used. Each teacher taught each 
treatment condition and all treatments were assigned to each classroom, 
but a given student was randomly assigned to only one treatment. 

Study II Results . The statistical analyses for Study II were planned 
in response to two research objectives: 

- To determine what student learning outcomes were affected by vari- 
ations in the percentage of higher cognitive questions in discussions; 

- To determine what student learning outcomes are affected by presence 
or absence of discussions. 

The student outcome measures used in the study are listed in Table 3. 

They include measures of students' factual knowledge related to the 

ecology curriculum and students' ability to work with and extend this 

knowledge at higher cognitive levels. Student performance was measured 

in both written and oral form inasmuch as the treatment conditions 

emphasized oral discussion. Each student was administered a test battery 

before, iiranediately after, and/or two weeks after the treatments to 

determine the effects of the treatments on achievement and attitudes 
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Study II 
Composition of Squares 

SQUARE 1 



TEACHER 


StHdOL 7 


SCHOOL 


9AM* Class 13 


10:00AM Class 14 


12 :30PM Class 15 


2:00PM Class 16 


8 


frpatmpnt 1 
B=3 
G=3 


irca Linen t o 
B=3 
G=3 


Ireatment 4 
8=5 

G=6 


Treatment 2 
8=3 
G=3 


7 


Treatment 3 
B=3 
G=3 


Treatment 2 
B=3 
G=3 


Treatment 1 
B=3 
G=3 


Treatment 4 
8=5 
G=4 


10 


Treatment 4 
B=2 
G=ll 


Treatment 1 
B=3 
G=3 


Treatment 2 
B=3 
G=3 


Treatment 3 
8=3 
G=3 


12 


Treatment 2 
B=3 
G=3 


Treatment 4 
B=6 
G=9 


Treatment 3 
8=3 
G=3 


Treatment 1 
8=3 
G=3 



ECOLOGY 
TEACHER 


SCH( 
9AM Class 17 


DOL 9 

10:00AM Class 18 


SCH 


OOL 10 


4 


Treatment 1 
8=3 
G=3 


Treatment 3 
8=3 
G=3 


12:30PM Class 19 
Treatment 2 
B=3 
G=3 


2:00PM Class 20 
Treatment 4 
8=8 
G=7 


11 


Treatment 3 
8=3 
G=3 


Treatrrsnt 2 
8=3 
G=3 


Treatment 4 
8=8 
G=7 


Treatment 1 
8=3 
G=3 


1 


Treatment 4 
8=6 
G=6 


Treatment 1 
B=3 
G=3 


Treatment 3 
8=3 
G=3 


Treatment 2 
8=3 
G=3 


5 


Treatment 2 
8=3 
G=3 


Treatment 4 
8=3 
G=7 

SQUARE 


Treatment 1 
8=3 
G=3 


Treatment 3 
8=3 

1 G=3 


ECOLOGY 
TEACHER 


SCHO 
yAM Class 21 


UL II 

10:00AM Class 22 


: 3 

SCHC 


JOL 12 


6 


Treatment 1 
8=3 
G=3 


Treatment 3 
8=3 
G=3 


12:30PM Class 23 
Treatment 4 
8=7 
G=7 


i;:OOPM Class 24 
Treatment 2 
8=3 
G=3 


2 


Treatment 3 
8=3 
G=3 


Treatment 2 
8=3 
G=3 


Treatment 1 
8=3 
G=3 


Treatment 4 
8=6 
G=8 


3 


Treatment 2 
8=3 
G=3 


Treatment 4 
8=6 
G=7 


Treatment 3 
8=3 
G=3 


Treatment 1 
8=3 
G=3 


9 


Treatment 4 
8=7 
G=7 


Treatment 1 
8=3 
G=3 


Treatment 2 
8=3 
G=3 


Treatment 3 
8=3 
G=3 



* Times are approximate. 



Treatment 1 
Treatment 2 
Treatment 3 
Treatment 4 
B = Boys 
^ = Girls 

ERIC 



= 25% Higher Cognitive Questions 
= 50% Higher Cognitive Questions 
= 75% Higher Cognitive Questions 
= Art Activity 
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TABLE 3 
Study II 

Achievement Measures 



INSTRUMENT 


VARIABLES MEASURED 


POini OF 
ADMINIS- 
TRATION 


APPROXIMATE 
TESTING 
TTMP** 


Comprehensive Tests. of 
Basic Skills-Reading 


Vocabulary; Comprehension 


Pre* 


52 minutes 


Ecology Information 
Test 


Amount of information about 
ecology 


Pre, post, 
delayed . 


li: minutes 


ura 1 1 es t 


no I M xy xo sxaxe ora iiy 
opinions, predictions, 
solutions, inferences, etc. 


pre, post 


in ITlTnil'^OC 

1 U lill ilii 

1 


Essay Test 

• 


Ability to state in writing 
opinions, predictions, 
solutions, inferences, etc. 

Ability to state in writing; 
reasons and if-thiin 
relationships. 


Pre, post 


25 minutes 


Population Test 


Ability to state in v;riting 
opinions,, predictions, 
solutions, inferences, etc. 
Ability to state in v;riting 
reasons ?Pd if-then 
relationships. 


Delayed 


25 minutes 


Quest ion- Gene rating 
Test: Paper-and- 
Pencil Measure 


Ability to generate 
questions. Quality of 
questions generated. 


Pre, post 


20 minutes 


Ques t i on- Gene ra t i ng 
Test: Oral Measure 


Ability to generate 
questions. Quality of 
questions generated. 


Pre, post 


2 minutes 



* Data collected by the participating school district, not by the researchers. 



*♦ These arc avcraqo times for test .administration. All tests except the 
Question Generating Test (Paper-and-Pencil Measure) were primarily power tests. 
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related to the specific curriculum which -;hey nave studied. The oral 
measures were administered to each student on a one-to-one basis and 
the students* responses were audiorecorded. These audiotapes were 
scored to obtain the student's rating for the specified variables. 
For purposes of this discussion, only the achievement outcomes will be 
considered since few differences occurred in the affective areas. 
Students appears to have liked all the treatments inasmuch as each pro- 
vided a diversion from regular classroom instruction. Complete infor- 
mation on all the variables may be obtained from the final report of 
the study. 

Table 4 presents the results of the analysis of variance for the 
ability, achievement, and attendance variables. The table contains the 
following information: 

- The name of each dependent variable. 

- The name of the variable used to adjust the student's scores on the 
dependent variable, if any. 

- The error mean square and its degrees of freedom. 

- The F-statistics for each dependent variable for treatment effects, 
class effects, teacher effects, square effects, and treatment by 
square interaction. 

- The omega squared (strength of association) statistics. This 
statistic is interpreted as the percentage of variance in the 
dependent variable attributable to the treatment effect for that 
column. 

Inspection of Table 4 indicates that there were six dependent 
variables on which the treatment group means differed significantly: 
Ecology Information Test Intentional Scale II, Ecology Information test 
25% Intentional Scale, Ecology Informatin Test Incidental Scale II, 
Oral Test Content Scale, Oral Test Logical Extension Scale, Essay Test 
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Content Scale. For these variables, the differences among treatments 
were greater than would be expected to occur by chance in 95 cut of 100 
comparisons of the treatments. 

Since all the information test subscal2s emphasize factual knowledge 
and the content scales of the oral and essay tests, while requiring higher 
cognitive responses, build these responses from the actual cOiitent of the 
questions asked during the discussions, the treatments seem to have had 
more influence upon recall of information than upon the various higher 
level outcomes. Of the six variables where statistically significant 
differences occurred, only the logical extension measure on the oral test 
represents a higher cognitive outcome that requires the student to formu- 
late responses that go beyond the concept of the treatments. 

To determine which treatment means differed significantly from each 
other on the achievement variables, planned comparisons were made of the 
treatment means. Table 5 presents these comparisons. 

The F-statistics for the planned comparisons (see the last 4 columns 
of the table) show that percentage of higher cognitive questions was a 
statistically significant influence on the amount of information acquired 
by students as measured by subscales on the Ecology Information Test, 
There also were some differences on the oral test measures, particularly 
when the art activity was compared with the combination of all the 
discussion treatments. 

The pattern of treatment mean scores for the subscales of the 
Ecology Information Test is depicted in Figure 2. It appears that a U 
cuv^ve describes the relationship between percentage of higher cognitive 
questions and achievement. In all cases the 50% Higher Cognitive Question 
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FIGURE 2 

Patterns of Treatment nifferences for 
Ecoloqy Information Subtests in Studv II 
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(HCQ) treatment has considerably lower outcomes than the other two discussion 
treatments. The 75% HCQ and 25% HCQ treatment outcomes fall at similar 
points for the various subtests. Outcomes for the art activity treatment 
approximate those for the 50% HCQ treatment. 

Since this finding was somewhat unexpected--a gradual increase in 
outcomes was expected moving from the art activity to 25% HCQ, to 50% 
HCQ, with 75% HCQ being highest— a secondary analysis of the data was 
requested. Dr. Richard Snow of Stanford University conducted this analysis.* 

The purpose for reanalysis of the data from Study II was to extend 
the understanding of the data from the Gall analysis rather than to double 
check the earlier analysis. 

A first step in the reanalysis was a factor analysis of the pre- 
measures for both Studies I and II. This showed five factors: (1) 
general scholastic ability and prior achievement (verbal, comprehension, 
and essay tests); (2) oral measures and discussion-attitude measures; 
(3) word association; (4) question generating, specific; and (5) quality 
ratings (question generating and oral). 

Simple and multiple regression analyses using the pre-factors as 
aptitudes predicted outcome variables with substantial multiple correlations. 
Some ou- uome measures were highly predictable such as the ecology infor- 
mation test and the essay test. Others, such as the question-generating 
test, were less predictable. 

These analyses also uncovered various higher order interactions. 
Some examples of aptitude- treatment interactions frcr. Study II include: 



* For complete information on the secondary analyses see Snow, et al.. 
Extended Analysis of Two Experiments on Teaching . Palo Alto: Stanford 
University, 1975. 
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- Based upon examination of the regression of the ecology information 
test measures on general ability (factor 1), it appears that the 
V-shaped average curves (see Figure 2) apply mainly to students of 
low ability. The 50% higher cognitive questioning (HCQ) condition 
is particularly bad for low ability students, while the 25% and 75% 
conditions are much better. Among higher ability students the 
three questioning conditions do not differ. 

- On the oral outcome measures, students low in general ability did 
best in the 25% HCQ condition while high ability students did poorest 
here. For them the 50% HCQ condition was the best. 

- Based upon a higher order interaction combining pre-factor 1 (general 
ability) and pre-factor 5 (quality rating) in predicting population- 
total measures, the 25% HCQ treatment was best for highs on these 
combined factors while the 75% HCQ treatment was best for lows. 

In general, the reanalyses showed that the effects of treatment in 
Study II were far more complex than might have been expected. Outcome 
is often a function of entering aptitudes and their interactions with 
treatment conditions. While the complex aptitude-treatment interactions 
can be interpreted only with caution, they suggest that no one treatment 
is routinely best for all students. 

Implications of the Study II Findings . The general conclusions to 
be drawn from the Study II that were reported here are obvious.* Fore- 
most is the rindings that a teacher's use of higher cognitive questions 
may not, in and of itself, lead to improved performance for all students. 
The context of the situation in which the questions are used v/arrants as 
much, if not more, consideration than the skill of asking higher cognitive 
questions. In particular, attention should be given to: 

- The structure of the discussion in which the questions are used. 
In Study II both the 25% HCQ and the 75% HCQ discussions had an 
obvious purpose and structure. In the 25% discussions, students 
were asked to devote most of the time to review of facts contained 
in the ecology unit curricular materials and a smaller amount of 
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* The findings in this paper do not include all the results obtained 
from Study II. See the final report for a complete statement of 
results. 
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time at the end of the discussion relating and extending these facts. 
The 75:^ discussions followed an opposite structure—a little time 
for review, much of the time relating and extending the ideas. The 
50% HCQ discussions, on the other hand, tended to mix up the two 
processes, and may, therefore, have confused some students. 

- The entering abilities of the students . The aptitude-treatment 
interactions discussed earlier in this paper underline, once again, 
the need to adapt instructional strategies to the needs of students. 
Appropriate use of a teaching skill— in this instance asking higher 
cognitive questions, with students with one level of entering 
aptitude—is not necessarily appropriate for other students. This 
conclusion was also supported in Study I where, using the ecology 
information test as criteria, students high in general ability did 
best when teachers used probing and redirection to follow-up student 
responses during a discussion. Low ability students were better 
off without probing and redirection. 

Effective teaching, therefore, probably centers apound when a skill 
is used, with whom, for what purpose, within what form of total instruc- 
tional situation rather than presence or absence of the skill. 

The implications of such complex research findings— and they have 
been reported in several recent research reports— are considerable for 
both the trainer of preservice teachers and the inservice teachers who 
are already in "daily" contact with students. 

In a competency-based preservice program that was responsive to the 
findings of Study II, a training program on the use of higher cognitive 
questions would include experience in generating and asking different 
types of higher cognitive questions, as most programs currently do. But 
it also would build knowledge about the types of student entry abilities 
and the degree of prior knowledge that should be considered when a 
discussion was planned and provide experience in recognizing and/or 
measuring various levels of these abilities. It would provide practice 
in planning and conducting discussions with different overall structures 
and sequences of questions with different students and require the 

24 
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trainees to analyze which discussion context seemed to work best for them 
with which students. 

A critical question is how many of the latter four training activities 
are included in the existing preservice training programs? Does work with 
the competency of asking higher cognitive questions include training in 
all the elements that interact when the skill is used effectively in 
teaching? 

For the inservice teacher, response to such findings may emphasize 
self-analysis and on-the-job practice. The inservice teacher might 
inquire into the structure of the discussions held in the classroom. 
Are the types of questions asked and the sequence in which they are asked 
modified according to the entry ability and prior knowledge of the students 
who are participating? 

On the other hand, inservice teachers also may need additional train- 
ing in order to apply findings that are as complex as those from Study II 
inasmuch as their previous training and experience may not have focused 
upon such elements as those student entry-abilities related to performance 
in various forms of discussion treatments. 

Incorporating research findings into teaching and teacher training 
may require more than reading the research report. A redesign of the 
training program may be necessary to incorporate findings into its structure 
and thereby affect existing competencies. 
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A Study of The Teacher as Math Tutor 

The Teacher as Math Tutor Study* also was conducted as part of the 
lTEP effort. This study was conducted with fourth and fifth grade students 
to answer questions concerning the effect of tutoring in mathematics upon 
students' mathematics achievement, self-concept in mathematics, attitude 
toward mathematics, and external locus of control as it relates to mathe- 
matics. The study was conducted by Dr. Barbara Ivory Williams of the Far 
West Laboratory staff. 

The study included four treatment conditions: two tutoring treat- 
ments, one in which teachers received special training in tutoring and 
one in which teachers received no special training; a supplementary 
treatment in which teachers conducted no regular tutoring but gave one- 
half hour of additional group mathematics instruction to the lower one- 
half of their classes each week for ten weeks; and a special control 
treatment in which teachers conducted their regular mathematics instruction. 

The Math Tutoring Study represents the third approach to the study 
of teaching outlined earlier in this paper. It is a familiar form of 
research on teaching which trains teachers to use a particular set of 
skills, then studies the effects in a given content area. 

Students in the study were designated as target students--four students 
from each classroom who were below tne school median on their Comprehensive 
Tests of Basic Skills (CTBS) score and their Iowa Tests of Basic Skills 
(ITBS) Modern Mathematics Supplement score and who received tutoring in 
the two tutoring treatments or received the extra half-hour per week of 
group; and non-target students— four additional students, matched on 

* For complete report of the study see: Williams, et al.. Math Tutoring 
Study: Teachers as Tutors ; Paraprofessionals as Tutors; Cross-age Tutoring . 
San Francisco: Far West Laboratory for Educational Research and Development, 
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CTBS pre scores, from each of the classrooms. Nontarget pupils were not 
identified to their teachers. Target/nontarget designations were not 
made for pupils in the special control treatment. Tables 6 and 7 indicate 
the number of teachers and students in each treatment by school for fourth 
and fifth grade. 

Based on this design, it was possible to consider the effects upon 
students that could be related to the tutoring process itself as con- 
trasted with those that were related to extra time spent in mathematics 
regardless of the teaching-learning process employed. 

In order to study the effects of tutoring, it was important that 
the treatment conditions be in operation over a reasonable period of 
time. For purposes of this study, two and one-half months— extending from 
early January to mid-March— were selected-. During this time, all teachers 
who participated kept a daily log indicating the student who was tutored 
(if appropriate to the treatment), the mathematics area in which the 
tutoring or extra group instruction occurred, the activity in which the 
remaining students in the classroom were engaged, the ndthematics area 
on which the class was working for the present week, an estimation of 
the progress that the tutored pupil made (if appropriate), the total 
number of minutes the tutoring session or the extra group session lasted. 

Tables 8 and 9 present data regarding the average number of minutes 
each target pupil was tutored per week. Since teachers in the supple- 
mental treatment gave no students regular, systematic one-to-one assistance 
and the tutoring sessions, for the most part, came close to the prescribed 
30 minutes of tutoring per week for each target student, the treatments 
appear to have been sufficiently well maintained to permit investigation 
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Average Number of Minutes Tutored Per Week 
by Teacher and Pupil 

TABLE 8 
Grade 4 





Hinicourse 5 Trained Tutors 


Regular Tutors 


School 


Pupil 


Pupil 
2 


Pupil 
3 


Pupil 
4 


Pupil 
1 


Pupil Pupil 1 Pupil 
2 3 I 4 


• 

3 
4 

A 

10 
11 


22.8 
19.5 
19.5 
18.0 
31.8 
16.7 


27.8 
18.0 
24.4 
17.0 
29.0 
18.3 


25.2 
16.8 
20.6 
21.0 
26.8 
28.7 


25.6 

21.8 

18.4 

21.0. 

25.7 

19.7 


29.5 
25.8 
22.8 
28.5 
27.5 
24.0 


29.5 [ 

26.7 ' 25.0 

i 

23.0 [ 21.2 
28.0 29.5 
25.0 27.5 
24.0 27.0 


26.7 
24.3 
29.5 
30.8 
30.0 




TABLE 9 . 
Grade 5 




Mini course 5 Trained Tutors 


Regular Tutors 




School 


Pupil 
1 


Pupil 
2 


Pupil 1 Pupil 
3 ! 4 


Pupil 
1 


Pupil 
2 


Pupil 
3 


Pupil 
4 


1 
3 
4 

9 

10 

" 1 


18.3 
22.9 
21.0 
24.6 
21.7 
29.0 


16.3 
25.7 
16.5 
21.4 
20.0 
28.5 


21.1 
23.7 
15.0 
19.1 
16.7 
22.5 


22.2 
21.9 

17.1 
23.3 
27.5 


21.0 
31.3 
27.5 
27.7 
15.0 
28.1 


27.0 

22.7 

24.4 • 

25.0 

22.5 

28.8 


24'. 0 
25.0 
24.4 
26.0 
25.5 
30.0 


24.0 
23.9 
21.9 
21.0 
18.0 
30.0 
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Teacher as Math Tutor Results . Before presenting the results of 
this study, it should be noted that within each school that participated 
in the study, the fourth and fifth grade students were randomly assigned 
to teachers for the school year in which the study was conducted. Such 
sampling procedures are rarely found in studies of teaching. They greatly 
increase the degree of general izability of the study results. 

The statistical analyses for the Math Study were planned in response 
to five research questions: 

- Do pupils who receive systematic mathematics tutoring perform 
better on a set of dependent variables than pupils of comparable 
ability in the • classes who do not receive systematic mathe- 
matics tutoring? 

- Do pupils who receive systematic mathematics tutoring from tutors 
trained in tutorial skills perform better on a set of dependent 
variables than pupils of comparable ability who receive systematic 
mathematics tutoring from tutors who have not had mathematics 
tutoring training? 

- Do pupils who receive thirty minutes of systematic mathematics 
tutoring per week perform better on a set of dependent variables 
than pupils of comparable ability who receive an additional thirty 
minutes of group mathematics instruction per week? 

- Do pupils who do not receive systematic mathematics tutoring 
but are in classes where systematic mathematics tutoring takes 
place perform better on a set of dependent variables than pupils 
of comparable ability who do not receive tutoring and are in 
classes where no systematic mathematics tutoring takes place? 

- Are there particular combinations of tutoring skills or inter- 
personal behaviors which are positively related to specific 
pupil outcomes? 

For purposes of this portion of the paper, we will consider only 
the first four questions. The fifth question will be discussed later 
when the Experimental Teaching Unit is presented. 
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The student outcome measures used in the study are listed In Table 10. 
They include measures of students' mathematics achievement, attitude toward 
mathematics, and external locus of control. All instruments were admin- 
istered from October 29 to November 9 in 1973 and again from March 18 to 
March 29, 1974. 



TABLE 10 

Math Tutoring Study 
Student Measures 



Instrument 


Variable Measured 


Point of 
Administration 


Comprehensive Tests of 
Basic Skills, Form Q, 
LevGl 2 


Computational 
Skills 


Pre, Post 


Modern Mathematics 
Supplement, Iowa 
Tests of Basic Skills 


Understanding of 
mathematical concepts 


Pre, Post 


Modified Sears 
Self-Concept Inventory 


Academic self concept, 
mathematics self-concept 


Pre, Post 


Dutton-Likert Attitude 
Towards Mathematics 
Scale 


Attitude toward 
mathematics 


Pre, Post 


Feelings About 
Mathematics Scale 


External locus of 
control as related 
to mathematics 


Pre, Post 



For purposes of this discussion, only the achievement outcomes will 
be presented. Complete information on all the variables may be obtained 
from the final report of the study. 

Table 11 presents the results of the analyses of covariance for CTBS 
for grades 4 and 5 for all treatment groups. The Table also contains the 
pre, post and adjusted mean CTBS scores for students in the various treat- 
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Inspection of Table 11 shows that differences among the seven treat- 
ment means approach traditional levels of statistical significance— that 
is an F-probabil ity of .05 or less. Looking at the adjusted post score 
means for each treatment for fourth grade, the nontarget students in the 
classes of the teachers who received special training in tutoring had 
the lowest mean score while the nontarget students in the classes of 
teachers who tutored but received no special training had the highest 
scores. At the fifth grade level, the special control students had the 
lowest adjusted mean post scores while the nontarget students in the 
trained teachers' classes had the highest scores. 

The ITBS findings showed significant differences at fourth but not 
fifth grade. In the fourth grade, the special control students showed 
the lowest adjusted post scores; the nontarget students in classes where 
teachers tutored but received no special training received the highest 
scores. The pattern for fifth grade, while not significant, was similar 
to that for the CTBS; the special control students were lowest, the non- 
target students from regular teacher tutor classes and the supplementary 
treatment target students were highest. 

Based on these results, it appears that some extra mathematics 
instruction may be helpful but it does not matter whether it is in the 
form of small group instruction or one-to-one tutorial assistance. Further, 
to have teachers who were providing extra instruction to some students 
who were below the school median on mathematics achievement may have had 
some serendipitous effects on other below-median students. 

Table 13 presents a three-factor analysis of covariance that permits 
more indepth answers to the research questions because important interactions 
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among variables could be accounted for. Factor one, method, contains the 
three primary treatments; the special control group was not used for this 
analysis. Factor two represents target nested within method and contains 
the target and nontarget designation of students. Factor three consists 
of the six schools in which the three primary treatment conditions were 
represented at both grade levels. 

For this analysis, only the ITBS results are considered. The majority 
of the variance in students' adjusted mean post scores was accounted for 
by their performance on the pretest. However, for grade 4 all three 
main effects— method, target/nontarget designation, and school— approached 
. conventional levels of significance. The method-by-school interaction 
also approached this level. 

With respect to the grade 4 method effect, the adjusted mean was 
highest for students in the regular (no special training) tutorial treat- 
ment and lowest for students in the trained tutorial treatment. The 
tarqet-wi thin-method effects can be explained by differences between 
adjusted means for the nontarget students in the trained teachers' classes 
and the nontarget students in untrained teachers' classes. School effects 
show that students in school 1 and school 9 differ by a greater amount 
than students in other schools. 

Based upon both the fourth and fifth grade analyses, it appears that 
target students did not perform markedly better than did non-target 
students, nor did 30 minutes' additional instruction in a one-to-one 
tutorial setting result in better student achievement than 30 minutes 
extra spent in group instruction. 

All the analyses presented above, in effect, compare not only the 
effects of tutorinq and no tutorina but al 
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Implications of the Math Tutor Study Findings . The implications of 
the Math Tutor Study results for both inservice teachers and teacher 
trainers, in our opinion, are three-fold. 

First, thoughtful reconsideration should be given to the previous 
research on tutoring. When positive effects were reported, was the 
additional learning time associated with the tutorial experience controlled 
for in the study? If not, the research findings, and possibly the use of 
teacher time for one-to-one tutoring, should be questioned. 

Second, based on the findings of the Math Tutoring Study, it appears 
that teachers i. ve some options when working with students who are below 
the median in mathematics achievement. Additional mathematics instruction 
is helpful for these students. But, it is at the discretion of the teacher, 
whether this is small group or one-to-onre instruction (tutoring as defined 
in the Math Study). Given the organizational and management problems 
faced by the teacher in a self-contained classroom who attempts to tutor 
individual students, such options are important. Determining which students 
will progress through additional small group instruction and which, if 
any, require one-to-one tutoring, then, becomes an important decision for 
the teacher. Once again, teaching and teacher training becomes a multi- 
faceted process that must attend to multiple student variables and multiple 
teaching strategies. 

Third, given that tutoring by specially trained' and untrained 
experienced teachers resulted in similar outcomes for students, questions 
should be asked about the training needs of experienced teachers. Based 
upon the tutoring skills contained in the Minicourse used to train the 
teachers in the Math Tutoring Study, the trained teachers appear to have 
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had considerably more knowledge about when to use the skills than did 
the untrained teachers. But they used the skills in a tutoring situation 
in approximately the same way as the untrained teachers. 

The measure of teacher knowledge about when to use tutoring skills 
was obtained through the use of a questionnaire that contained 32 controlled 
stimulus situations which presented specific mathematics learning problems 
followed by at least one question about the situation and a blank space 
for recording the response. The means and standard deviations for the 
teacher sample are reported in Table 14. 

Teacher use of certain tutoring skills and affective behaviors was 
measured through the use of an observation form. Each teacher, trained 
or untrained, was observed at least six times. The group means for each 
skill are reported in Table 15. 

The similarity in the average number of times most of the behaviors 
occurred in a tutoring session regardless of whether the teacher did, or 
did not, receive special training raises a significant issue for those 
involved in inservice teacher training. This is the issue of what form 
of skill training is most productive for experienced teachers. Should 
the training focus upon presentation of teaching models and practice of 
these models? Or, would the teacher develop more insight into the use 
of the skills by being placed in an instructional situation that necessitates 
and/or implies that certain skills be used, then observing to determine 
that the skills were used and when, and noting the resulting student 
responses? Or some combination of both? 

It is clear that in this instance, thought needs to be given to 
differentiating the training of pre-service teachers and practicing 
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TABLE 15 

Observation Variable Means for Minicburse 5 and 
Regular Tutors and t-test for Difference in the Means 



Vai 1 dU 1 c 


Tra i ned 
1 eacner 
Tutor 


Means 
untrainea 
Teac'her-Tutor 


era 




D1ag. Q. 


13.36 


8.86 






Prompting 


29 AZ 


28 63 






Eval. P. 


.64 


.56 




Ql 


Prac. P. 


.48 


23 


• U/ 7 


Om 1 O 


Demo (t) 


47.77 


26 Q5 


CO . ooo 


A7 


Demo (f) 


2.51 


2.58 


.7111 


-.10 


Praise 


13.72 


11.59 


1.449 


1.47 


Rapport 


5.71 


5.73 


.194 


-.10 


Mot. Stmt. 


.61 


.68 


.076 


-.92 


Decl. Stmt. 


14.23 


13.84 


2.491 


.16 


Criticism 


1.55 


2.15 


.414 


-1.45 


Recrim. 


.21 


.31 


.220 


-.45 



Standard error of the difference between the sample means. 

t statistic distributed on 29 degrees of freedom (Observations 
on Minicourse 5 and 16 Regular Tutors). 



* Significant at .05 level. 
** Significant at .01 level. 
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teachers. Based on our findings, it would appear that the experienced 
teacher might already possess in his/her repertoire those skills that 
might be proposed as the basis for a training program. If so. attention 
to establishing criteria for entry level skills is of paramount impor- 
tance. Often, providing a teacher with precise descriptions of those 
skills in which training is proposed and a way to observe and analyze 
his/her teaching in relation to these is all that is necessary to place 
them in operation. On the other hand, a pre-service teacher will probably 
need to attend more to the actual development of the skills. 
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Applying Research Methodology to Teaching 

As stated earlier, research findings more frequently have been applied to 
teaching and to the training of teachers than has the methodology which is 
used in educational research. However, in the few instances where methodology 
has been adapted, it has served successfully to train teachers and quite 
often has become a part of the everyday operations of teaching. Both micro- 
teaching and interaction nalysis are illustrative of research methodologies 
that were developed primarily as data collection procedures, and later were 
adapted to teacher training systems. Today, as regular practice, teachers can 
occasionally be observed using both to monitor and to analyze their own 
teaching. 

Two examples of such methodology which grows out of ETEP research will 
be proposed here as potentially significant for the training of teachers. 
These are (1) the use of Experimental Teaching Units (ETU), and (2) the 
use of semi programmed teaching units. 

In this discussion, no attempt is made to distinguish between the pre- 
service and inservice training of teachers. Instead, however, it is recognized 
that entry levels into either of the examples described here will differ de- 
pending on the experience of the trainee. It is also assumed that the two 
procedures can be adapted to meet the needs of either a preservice teacher 
training program or an inservice teacher training program. 

Experimental Teaching Units (ETU) . The ETUs were developed in order 
to hold constant the curriculum being taught by a sample of teachers being 
studied. At the Far West Laboratory, they were used initially in the Math 
Tutoring Study of ETEP, and the technique was adopted in the Beginning 
Teacher Evaluation Study (BTES) for the California Commission for Teacher 
Preparation and Licensing. From this work, five ETUs have bean developed^ 
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three for the teaching of mathematics (one at the second grade, one at the 
fourth grade, and one at the fifth grade), and two for the teaching of reading 
(one each at the second and fifth grades). 

Essentially, an ETU consists of an introduction to the teacher which 
discusses the rationale for the unit; specific performance objectives, each 
of which are keyed to items on the pre- and posttests; pretests for the 
students; a wide variety of instructional materials and activities from 
which teachers may choose; and a posttest for students. Teachers are in- 
structed to select those objectives on which they will focus and the in- 
structional materials and activities they will use. Pretest information 
for their students is supplied. 

Teachers then may be directed to teach the ETU in any style they prefer 
to as many students as they desire using any of the instructional materials 
and activities and/or they may be asked to teach all objectives to all 
students, differentiating instruction based on the student's performance on 
the pretest (i.e., according to whether the student has already mastered 
the objective and therefore is ready to extend the concepts and ideas, 
or has not mastered the objectives). 

The duration of an ETU is approximately three weeks, and teachers 
utilize the regular instructional time set aside each day for the content 
area covered by the ETU. Although an ETU focuses on content areas that 
conform to accepted curriculum objectives for the given grade level, it 
attempts to cover material not ordinarily stressed by teachers at that 
grade level. In this way, the teaching experience, as well as the resultant ' 
learning experience, is more likely to be unfamiliar. This, in turn, 
maximizes the conditions under which the effects of teaching can be 
observed. 

The usefulness of ETUs for studying teaching is quickly apparent. 
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They appear to make it possible for the effects of teaching to show more 
dramatically than do other approaches to studying teaching, and they there- 
fore appear to have value as tools for studying effective teach-' ng. 

In both educational research as well as in the public scho( ^ p -actices 
of evaluating teachers, the effectiveness of teaching typically is jetermined 
on the basis of student performance on an achievement tests. It is thus 
hypothesized that if teachers are teaching effectively, their students will 
show gains in a pretest-posttest (beginning - end-of-year) comparison. While 
this practice predominates, there is growing evidence that the items on such 
achievement tests not only do not focus specifically enough on the in- 
structional objectives a teacher he 3 set for his/her students, but they 
appear to be biased toward the socio-economic status or the cultural realities 
of many students. Thus, it would appear that the use of such procedures 
is questionable as a means for determining teacher effectiveness. 

The ETU, on the other hand, by focusing on specific instructional 
objectives, and by using criterion pretest and posttest items tied to these 
objectives, makes it possible to study relationships between teacher 
characteristics, behaviors, instructional decisions and moves, etc. and 
student outcomes. 

Working with an ETU, instructional decisions made fay a teacher may be 
monitored. Attention also may be given to questions such as what objectives 
were selected? which students received the instruction? was such in- 
struction appropriate? did the students already meet the criteria set 
by the pretest for those objectives? what instructional materials and/or 
activities were utilized? did these activities teach to the instructional 
objectives? 

In the Math Tutoring Study of ETEP, the subsection of the study in 



45 



which the ETU was used showed the advantages of this methodology over the 
year-long techniques for determining teacher effectiveness. 

In the Math Tutoring Study, the relationships between teacher use of 
specified tutoring skills and other behaviors were studied in terms of 
students' year-long mathematics achievement in am ETU. For purposes of 
these analyses, the variables of interest were clustered a priori into 
five major clusters as outlined in Table 16. Cluster A represents students' 
prescores on the various measu.-es used in the study.* Clusters B, C, and 
D represent the teaching behaviors which were observed in the study. Cluster 
E includes the treatment conditions which we discussed earlier in this paper. 
The only difference in the year-long and ETU analyses was the addition of 
the students' ETU prescores to Cluster A. 

While the Math Study report includes several analyses** which in- 
corporate the clusters as outlined in Table 16, we have selected the 
communal ity analyses for Grade 4 to illustrate our thesis regarding the 
usefulness of the ETU. The primary focus of the communal ity analyses is 
on the uniqueness of estimates of variance of each cluster. Clusters with 
large uniqueness estimates would be expected to influence student performance. 

Table 17 reports the results of the fourth grade analyses. Of 
particular interest in these results are the lowered uniqueness estimate 
for Cluster A and the higher uniqueness estimates for Clusters B, C, and D 
in the ETU analyses as compared with the year-long analyses. Apparently, 
student entry-level influences posttest performance less in an ETU than 
in year-long achievement measures. This, in turn, should make it possible 
for teacher effects to be observed. The increase in uniqueness estimates 

* Refer to Tabfe 10 for description of these measures. 

**See final report of A Study of The Teacher as Math Tu tor for complete 
information, ' — ' ^ 
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for the teacher behavior clusters suggests that this does, in fact, occur. 

Further evidence of the extent to which teacher effects appear during the 
teaching of an ETU as contrasted with year-long studies of teaching is provided 
by the three- factor analyses for the Math Study. Using Grade 4 analyses to 
illustrate (see Table 18), the treatment appears to have been more marked 
during the teaching of the ETU. At least few statistically significant 
differences were identified relative to long-term student gains on the 
CTBS compared with gains on the ETU. The method by school results are 
particularly relevant since, in the three-factor analysis, each cell 
represents a teacher. This, then, is a measure of teacher effect. 

The usefulness to the study of teaching of the methodological approach 
incorporated in the ETU is clear. This same methodology also can be applied 
successfully to the practice of teaching. This can be done in two ways. 
First, teachers can use the ETU format to structure and sequence instruction 
for their students and to observe and analyze their own instructional styles 
to identify what teaching strategies are most effective. In using ETUs in 
this way, the second value to teachers becomes apparent: ETUs can be used 
by teachers to shan the effect they are having on student learning. Both 
are discussed here. 

The idea of instructional units is not new to teaching. However, most 
instructional units now available focus upon much larger units of instruction 
than does an ETU. Used for teaching, each ETU would build around a limited 
number of instructional objectives which are uniquely related to each other 
in order to construct a specific concept and/or to bring together several 
small concepts into a larger understanding of a single topic. The sequencing 
of several ETUs could thus be perceived as teaching a concept or a unit of 
knowledge. 

It is perhaps easiest to discuss the use of an ETU for purposes of 
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teaching by describing what might be a typical scenario. Teachers at a 
given school site meet quite often to determine instructional goals for their 
grade level in all content areas. From these goals, they establish specific 
instructional objectives for their own students. Focusing on one content 
area, they could then construct a series of ETUs that, sequenced, would 
constitute teaching the objectives for that content area. For each ETU, 
teachers would specify several related instructional objectives and construct 
items for both a pretest and a posttest that would measure student performance. 
In addition, instructional materials would be collected and activities 
organized to teach the objectives. Finally, teachers would determine for 
themselves what teaching strategies to use while teaching the ETU. 

The ETU would be operationalized by pretesting the entire class in 
order to determine which students could already perform the tasks asked of 
them by the instructional objectives. For those students who could not, 
teachers would then follow their plan and teach the ETU, which would normally 
comprise twenty to forty minutes of instruction per lesson for a period of 
two to three weeks. At the end of this time, the posttest would be ad- 
ministered to determine the extent to which students who received in- 
struction could now perform the tasks required. 

Such an approach to teaching, of course, is exemplary of differentiation 
of instruction based on differences in both students' needs and abilities. 
In such a system, teachers would have to differ their instruction to 
accommodate the needs of both those who "know it" already and those who 
need instruction. The existence of a repertoire of ETUs which in sequence 
cover a content area would insure that this is possible. 

It is easy to see how teachers could use this approach to analyz'e 
their own teaching and to illustrate the effect that their teaching has on 
their students' learning. As discussed earlier, the use of the ETU seems 
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to be a more viable technique to show teaching effect than merely the use 
of student achievemeri test results. With the ETU, teachers are better 
able to tie test items to instructional objectives on which they plan to 
focus, to sc:;t criteria which constitute successful student achicve:T;;?r;t of 
these Ob; actives, and to measure their students' growth. In trro way, 
teachers would also be establishing the ways in which their own teaching 
is effecting student learning. 

Semi-prog raiTwed teaching units . The semi -programmed teaching unit 
originated in educational research as a way to hold a teaching strategy 
constant in order to study the effects on learning. Such units most 
frequently are used in experimental research designs. In ETEP, the notion 
of semi -programmed teaching units was applied in the studies of questioning, 

This type of teaching unit features as its central focus a script 
which is provided to the teacher for use in leading class/group discussion. 
The discussion is only "semi "-programmed because, although it provides the 
teacher with sequenced questions to ask, some of the teacher's behavior 
must remain dependent upon how students respond to the questions. For 
instance, the intent of such a semi-prograruned questioning script might be 
to build a concept by proceding through a series of questions that first 
lay the groundwork with fact questions, then apply principles by asking 
higher cognitive questions. However, should a student answer, "I don't 
know" to an initial question, the teacher by necessity would have to depart 
from the script long enough to establish with the class an answer to that 
question. Thus, a semi -programmed teaching unit can provide a useful guide 
for a teacher to follow in constructing concepts with his/her class, but 
a great deal of its success remains with a teacher's ability to augment 
the script when necessary. 

In the ETEP study on the effects of questioning on student achievement 
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and attitude, the semi -programmed discussion script usually consumed twenty 
to thirty minutes. Each discussion, regardless of treatment- condition, 
consisted of sixteen questions. The decision to use sixteen questions was 
based upon pilot work, which indicated that teachers in the cixth grade 
typically could ask fifteen to twenty questions in a twenty to thirty minute 
period without a time difficulty for either the teacher or the students. 
In Study II these sixteen questions varied in the number of fact, multi- 
fact, and higher cognitive .questions included in the script based on the 
percentage of higher cognitive questions in the treatment. The fact and 
multi-fact questions corresponded to Bloom's knowledge levels. The higher 
cognitive questions were based on the processes described for the upper 
levels of Bloom's cognitive taxonomy. 

Two criteria were used to generate the fact and multi-fact questions. 
First, each question required the statement of a fact (o.- facts). Second, 
the fact required was explicitly stated in the curriculum material for 
the same day's lesson in which the question was asked. No questions were 
repeated in different lessons. 

The higher cognitive questions also were constructed according to two 
criteria. First, each higher cognitive question required predictions, 
solutions, explanations, evidence, generalizations, interpretations, or 
opinions. Second, these predictions, solutions, etc. were not directly 
stated in the curriculum material but required the student to expand on 
or use the information presented in the day's lesson in a new way. The 
classification system used for the higher cognitive questions was as 
follows: 

- Analysis questions, those which elicit: 

• motives or causes of observed events; 

• inferences, interpretations, or generalizations; 

• evidence to support inferences, interpretations, generalizations. 
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- Synthesis Questions, those which elicit: 

• predictions; 

• solutions to problems; 

• original conmuni cations. 

- Evaluation Questions, those which elicit: 

• opinions about issues; 

• judgements about the validity of ideas; 

• judgements about the merit of problem solutions. 

The discussions were developed so that the relative proportions of each 
type of higher cognitive question in a lesson were balanced. Thus, "analysis 
"synthesis," and "evaluation" questions were approximately equally represented 
To insure that questions were relevant to the curriculum the curriculum 
objectives were used as a basis for constructing the questions. 

The results of the study, discussed earlier in this paper, underscore 
the importance of the semi -programmed approach to teacher-led" discussions 
in experimental settings. Besides the need to attend to the ratio of higher 
cognitive questions in relation to fact-recall questions in any discussion, 
teachers also should be aware that in the Questioning Studies students who 
were poor readers did as well on higher cognitive questions as did those 
students who were good readers so long as teachers first established the 
necessary information base. In the instance of Questioning Study II, 
semi -programmed discussion scripts for the 25% HCQ and 75% HCQ treatments 
organized the questions in such a way that fact-recall questions were 
presented early in the discussion so that necessary information was 
related to all the students, either by asking these questions of those who 
could, read the material in which the information was contained or by the 
teacher providing the answers to the questions. It is this sequencing of 
questions in the semi -programmed script— presenting fact-recall questions 
first to establish an information base, and then questions that direct 
students to analyze, synthesize, and evaluate— that establishes the potential 
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of the units for teachers and teacher training re questioning. 

It would thus apr ar that semi -programmed teaching units are promising 
both as a training device for teaching questioning skills and as a process 
by which teachers can pre-plan the construction of concepts and/or concept 
hierarchies to be developed with their students. As a training vehicle, 
such units organize the content for a given subject area around established 
instructional objectives, direct the teacher through an established series 
of activities with his/her- students , and provide a semi-progranmed dis- 
cussion script for the teacher to follow in leading students to understanding, 
to analyzing and synthesizing, and to evaluating what they have learned. 

The process of using the semi -programmed discussion script is, in itself, 
an important process for teachers to consider using. To date much of the 
training in inquiry approaches to teaching has focused on teacher behavior 
which probes, elicits, redirects, rewards, etc., a student, rather than on 
the process of constructing thoughtful, logical questions which properly 
sequenced can serve as a "map" that leads students to understandings and 
encourages them to enter into higher cognitive thought processes. Teachers 
can be taught to construct such scripts to use themselves with their 
students. Such scripts focus on phrasing, ahead of the discussion, 
well -constructed, clear questions and sequencing them in order to develop 
concepts. By additionally assigning students' names ahead of time to 
questions, teachers can consider the strengths each student may bring to 
such a discussion without eliminating the poorer readers from involvement 
in pursuing solutions to the higher cognitive questions. 
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A New Approach to Research and Development in Teaching 

The foregoing discussions of findings and methodology from the ETEP 
research on teaching underlines the need to apply all that happens in 
research to teaching and teacher training as it is happening rather than 
waiting until the research process is completed. When the current research 
and development model is followed, a great deal of educational research 
does not find its way into classroom application and a great deal eventually 
is found to be irrelevant to the improvement of teaching. 

Concern for» explanations of, and possible solutions to this problem 
have been voiced during the past year by several experts in the research 
end development field (e.g., Guba & Clark, Ovsiew, Krathwohl). Considerable 
discussion has centered around the inadequacies of the linear research and 
development model and the limitations of studing the effects of single 
teaching skills. 

There is a clear neec 'o d-velop strategies that will increase the 
likelihood that researcr on teaching wfll produce results that both can^ 
and will be applied in th^ r^lassrc jm tc improve education. For these reasons, 
we are proposing a new model for the conduct and dissemination of research 
on teaching. This model is designed to attack the problems of linearity 
in the concept of research and development and restrictions of single skill 
oriented research on teaching. We will deal here with only a brief des- 
cription of the model. More complete information can be obtained upon 
request.* 

In the traditional reserrch cycle, researchers, trainers, and class- 
room teachers have largely operated independently; the researchers have 
conducted their studies with verly little interaction with trainers and 

*For complete information see: Ward and Tikunoff: Draft Narrative Outline , 
Program of Research and Development in Teaching . San Francisco: Far West 
^ Laboratory for Educationl Research and Development 1975 
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teachers, only as absolutely necessary to conduct a study. And trainers 
and teachers have seldom provided feedback to the researchers regarding 
the relevance and applicability of their methodology and findings. See 
Figure 3 below. 



TTgURe 3 

Traditional R&D Cycle in Teaching 



RESEARCH 
(Discovery and 
confirmation 
studies) 



DEVELOPMENT 
(Findings 
engineered 
for use in 
practice) 



DISSEMINATION 
(Training of 
users) 



ADOPTION 

(Classroom 

application) 



Instead, it is proposed that an interactive model be developed, tested, 
and applied. See Figure 4 below. 
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In this new approach, the research has built into it a concern for 
training requirements, stemming from research techniques and research findings, 
and for classroom reality, as showin by the two-way arros in Figure 4. The 
research team, depicted in Figure 5, is composed of not only researchers but 
also trainers and classroom teachers themselves. 
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Researchers Devel opers 
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All participate in decisions regarding what should be studied and how 
it should be studied. Once this decision is reached, the researchers proceed 
to conduct the study. Concurrently, the trainers and developers examine 
the data collection procedures used in the research to establish ways for 
applying them in training, conduct small-scale pilot tests of some of these 
applications, hypothesize possible outcomes of the research, and implications 
for training. 

The teachers undertake informal and participant observation studies 
of the extent to which they, and possibly other teachers, are already 
utilizing those aspects of teaching on which the research is focused. 
This would include both positive and negative aspects of teaching and 
learning. 

The purpose of this inquiry by teachers, trainers, and developers 
would be to provide some guidelines regarding how much and what types of 
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training would be needed in order to implement the research findings in 
classrooms in general. 

Several important factors that must be considered in selecting the 
areas of teaching to be studied include: 

- Teaching does not occur as a separate activity. The purpose of 
teaching is learning on the part of students. Therefore, any 
study of teaching must take into account learning and thus the 
actions and characteristics of the learner(s) as well as the 
teacher(s). 

- Teaching and learning are not content free. If one is to be 
concerned about the multiple dimensions of teaching (and learn- 
ing) the content of what is to be learned (and taught) must be 
considered. 

- Within the interactive model the applicability of the aspect 
of teaching to be studied to regular classroom instruction 
should be considered from the earliest stages of discussion. 
Realistic and functional teaching events should be the focus 
of the research. 

The research methodology to be utilized in the proposed model should 
move beyond that traditionally applied in educational-psychology based 
research. That is not to say that th^ "traditional" methods of educational 
research would not be used. What would be done is to consider additional 
approaches. 

Our initial effort in applying anthropological-ethnographic pro- 
cedures to the study of teaching through the Beginning Teacher Evaluation 
Study* suggests one new research approach that has high potential for 
accommodating multi-dimensional, realistic studies of teaching. Others 
need to be identified. 

At the same time, the anthropological-ethnographic process has opened 
up a need to consider new avenues for collecting information about teaching. 
In particular, the use of the teacher as a contributor to basic knowledge 
aboift teaching— in other words as a researcher— while still maintaining 

*The Beginning Teacher Evaluation Study is a program of research on teaching 
Q also underway at the Far West Laboratory. 
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his/her teaching role should be tested. 

Such a function for the teacher is essential to the interactive R&D 
modeK Examples of the ways in which the teacher might perform as a 
researcher include: 

- Prepare ethnographic protocols delimiting the teaching/learning 
events that occurred, the types of students who were involved, 
the content that was learned, whenever an instructional lesson 
was particularly successful. These protocols, then, could be 
compiled, as a source of information abput what aspects of teach- 
ing warranted further study. 

- Conduct self-observation and/or observations of his/her peers 
to determine whether the dependent and independent variables 
proposed for a research study are realistic. 

One of the advantages of an interactive R&D irodel is the immediate 

application of both the data collection procedures i'nd the research 

hypotheses to training and classroom practice. Tha design of the delivery 

system for achieving this goel must consider initial and later applications 

of the procedures and findings, and should include on the research team 

the following team members: 

- trainers who are actively involved in joth preservice and 
inservice professional devclooment; 

- developers who are open to muUiple approaches to dellv^ry^ 
i.e., not partial to use of "products" as the only viable 
training approach; and 

- teachers who are working in a setting where they have access 
to input from other teacher:;, e.g., be working in a teacher 
center program (complex). 

All should be recognised by their peers as being representative of their 
needs and interests. This is essential if the "acceptability of proof" 
issue is to be met. 

W-^'thin an interactive R&D system such \s we have proposed, the four 
approaches to application of research that were set forth in this paper can 
be operetionalized most expeditiously. 
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